Dental Image Feature Detection

ABSTRACT

A system includes a computing device that includes a memory configured to store instructions. The system also includes a processor to execute the instructions to perform operations that include receiving data representing one or more images of dental information associated with a patient. Operations include adjusting the data representing the one or more images of dental information into a predefined format, wherein adjusting the data includes adjusting one or more visual parameters associated with the one or more images of dental information. Operations include using a machine learning system to determine a confidence score for one or more portions of the one or more images of dental information, and producing a representation of the determined confidence scores to identify one or more detected features present in the one or more images of dental information.

RELATED APPLICATIONS

This application is a continuation application and claims priority toU.S. application Ser. No. 17/691,894, filed on Mar. 10, 2022, which is acontinuation application and claims priority to U.S. application Ser.No. 17/196,888, filed on Mar. 9, 2021, which is a continuationapplication and claims priority to U.S. application Ser. No. 16/387,388,filed on Apr. 17, 2019, which claims priority to U.S. ProvisionalApplication No. 62/658,675, filed on Apr. 17, 2018. The contents ofwhich are hereby incorporated by reference.

BACKGROUND

This description relates to using machine learning methods to analyzeand detect features, i.e. dental pathologies, in (dental) radiographs.

Dental radiographs are one diagnostic tool in dentistry. Dentists mayhave limited training in reading radiographs and little support frome.g. an additional radiological department, assisting them in theirdiagnosis. Due to such large volume of radiograph data and limitedanalysis time, false negative and false positive errors may occur andcould potentially lead to health risks and increased health costs due tomissed detection or false treatment.

SUMMARY

The described systems and techniques can aid dental clinicians in theirability to interpret dental images, including but not limited tointra-extra oral radiographic imaging (e.g. bitewing and periapicalradiographs), extra-oral radiographic imaging (e.g. panoramic x-ray),computed tomography scan (CT-scans) coming from a CT scanner, Positronemission tomography scan(PET-scans) coming from a Positron emissiontomography-computed tomography scanner and Magnetic resonance imaging(MRI) scans coming from a MRI scanner, to correctly identifypathological lesions. By highlighting the potential features ofinterest, including but not limited to potential suspicious radiolucentlesions and potential carious lesions (also called cavities) and otherpathological areas, the viewer of the radiograph can quickly recognizethese detected features to reduce the number of missed lesions (falsenegatives) and wrongly identified lesions (false positives). Byemploying machine learning techniques and systems to analyzeradiographs, which are presentable on displays, electronic or printedreports, etc., an evaluation of patient health condition can beefficiently provided, thereby allowing the dental professional to makean informed decision about treatment decisions. While many methodologiescan be employed for pathology detection in dentistry, artificialintelligence techniques, such as deep learning algorithms, can exploitsuch radiographs, the images information, for training and evaluation inan effective way. By developing such techniques, the diagnostic errorsin dentistry can be reduced, pathologies can be detected earlier, andthe health of the patients can be improved.

In one aspect, a computing device implemented method includes receivingdata representing one or more images of dental information associatedwith a patient. The method also includes adjusting the data representingthe one or more images of dental information into a predefined format.Adjusting the data includes adjusting one or more visual parametersassociated with the one or more images of dental information. The methodalso includes using a machine learning system to determine a confidencescore for one or more portions of the one or more images of dentalinformation, and producing a representation of the determined confidencescores to identify one or more detected features present in the one ormore images of dental information.

Implementations may include one or more of the following features. Themethod may further include transferring data representative of the oneor more images of dental information associated with the patient to oneor more networked computing devices for statistical analysis. Themachine learning system may employ a convolution neural network. Themachine learning may be trained with dental imagery and associatedannotations. One or more annotations may be produced for each of theimages of dental information. The one or more detected features mayinclude a radiolucent lesion or an opaque lesion. The producedrepresentation may include a graphical representation that ispresentable on a user interface of the computing device. The producedrepresentation may be used for a diagnosis and treatment plan. An alertor recommendation may be produced by using the produced representationfor the diagnosis and treatment plan.

In another aspect, a system includes a computing device that includes amemory configured to store instructions. The system also includes aprocessor to execute the instructions to perform operations that includereceiving data representing one or more images of dental informationassociated with a patient. Operations also include adjusting the datarepresenting the one or more images of dental information into apredefined format. Adjusting the data includes adjusting one or morevisual parameters associated with the one or more images of dentalinformation. Operations also include using a machine learning system todetermine a confidence score for one or more portions of the one or moreimages of dental information, and producing a representation of thedetermined confidence scores to identify one or more detected featurespresent in the one or more images of dental information.

Implementations may include one or more of the following features.Operations may further include transferring data representative of theone or more images of dental information associated with the patient toone or more networked computing devices for statistical analysis. Themachine learning system may employ a convolution neural network. Themachine learning may be trained with dental imagery and associatedannotations. One or more annotations may be produced for each of theimages of dental information. The one or more detected features mayinclude a radiolucent lesion or an opaque lesion. The producedrepresentation may include a graphical representation that ispresentable on a user interface of the computing device. The producedrepresentation may be used for a diagnosis and treatment plan. An alertor recommendation may be produced by using the produced representationfor the diagnosis and treatment plan.

In another aspect, one or more computer readable media storinginstructions that are executable by a processing device, and upon suchexecution cause the processing device to perform operations that includereceiving data representing one or more images of dental informationassociated with a patient. Operations also include adjusting the datarepresenting the one or more images of dental information into apredefined format. Adjusting the data includes adjusting one or morevisual parameters associated with the one or more images of dentalinformation. Operations also include using a machine learning system todetermine a confidence score for one or more portions of the one or moreimages of dental information, and producing a representation of thedetermined confidence scores to identify one or more detected featurespresent in the one or more images of dental information.

Implementations may include one or more of the following features.Operations may further include transferring data representative of theone or more images of dental information associated with the patient toone or more networked computing devices for statistical analysis. Themachine learning system may employ a convolution neural network. Themachine learning may be trained with dental imagery and associatedannotations. One or more annotations may be produced for each of theimages of dental information. The one or more detected features mayinclude a radiolucent lesion or an opaque lesion. The producedrepresentation may include a graphical representation that ispresentable on a user interface of the computing device. The producedrepresentation may be used for a diagnosis and treatment plan. An alertor recommendation may be produced by using the produced representationfor the diagnosis and treatment plan.

These and other aspects, features, and various combinations may beexpressed as methods, apparatus, systems, means for performingfunctions, program products, etc.

Other features and advantages will be apparent from the description andthe claims.

DESCRIPTION OF DRAWINGS

FIG. 1 illustrated the integration of the software into the work stationof a dental professional.

FIG. 2 illustrates functionalities of the artificial intelligence baseddetection system.

FIG. 3 is a block diagram of an internet-based computer network toprovide the detected features in dental imaging.

FIG. 4 a block diagram of the feature detector which manages andprovides the detection of features in dental imaging.

FIG. 5 illustrates the operations for training and inferencing thesystem for detecting features.

FIG. 6 illustrates an exemplary architecture of a neural network model.

FIG. 7 illustrates an exemplary network architecture for a convolutionalneural network.

FIG. 8 shows the instructions for a convolutional neural networkarchitecture.

FIGS. 9A, 9B, and 9C illustrate a data gathering, annotation andeducational system.

FIG. 10 illustrates a flow chart of operations executed by anidentifier.

FIG. 11 illustrates an example of a computing device and a mobilecomputing device that can be used to implement the techniques describedhere.

DETAILED DESCRIPTION

Referring to FIG. 1 , a dental analysis system 100 includes an imagingmachine, 102, e.g. an x-ray machine, which emits x-ray beams, 104, to anx-ray sensor, 106 (e.g., an intra-oral sensor, an extra-oral sensor,etc.) for taking radiographic images of the jaw and teeth of a patient.The x-ray sensor 106 is connected to a computing device (e.g., acomputer system 108) including a display 110 capable of presentingradiograph information for review and study for the user of the dentalanalysis system, including but not limited to dental professionals (e.g.general dentists, endodontists, maxilla-facial surgeons), hygienists andother radiologists. One or more techniques, formats, etc. may be used toprovide the radiographic data to the computer system 108; for example,the radiograph can be provided in a raw image data-format which will beprocessed by a sensor-specific software into digital imaging andcommunications in medicine (DICOM) or any other image format (tif, png,jpg etc.) by the computer system. The computer system 108 may alsoexecute operations so one or more artificial intelligence techniques canbe used to analyze this data and present results. In some environments,the computer system 108 can provide the data to other computing devices(e.g., a cloud computing system, service, etc.) to initiate a moredistributed processing of the data. The machine learning techniques andother processes of the data utilize the dental image and associateddental image information, e.g., the age and gender of the subject, i.e.the patient, when the image was taken and other image meta-data such asx-ray sensor and model used, and other potential DICOM tags which do notconstitute as personal health information. Once processed, analyzed datafrom the artificial intelligence techniques can be returned to thecomputer system 108 for presentation and review (e.g., by a dentalprofessional). The analyzed data of the dental image can be used in manyways: First, one or more presentation techniques may be employed by thecomputer system 108 to present the analyzed data; for example varioustypes of user interfaces as one exemplified in interface 112 on themonitor 110, graphical representations, etc., may be used to efficientlypresent the data and quickly alert the professional to potential areasof interest signaling on the monitor 110 potential detected featureswhich need immediate attention by the user. Detected features in thedental images may include radiolucent lesions, opaque lesions, otherpotential pathological lesions such as tooth-related radiolucentlesions, all types carious lesions, all kinds of periapical radiolucentlesions (including but not limited to cysts, infections etc.), bonefractures, tumors, osteonecrosis, other dental pathologies or obviousseemingly pathological radiolucent image parts and other features suchas teeth and teeth-position/numbering, missing teeth, wisdom teeth,crowns, bridges, implants, and other anatomical characteristics such asbone density, height, width of the bones and angles, positioning,distances etc. between different facial structures (e.g. sinuses),tissues, gum and bone structures (e.g. implant and other treatmentplanning), margin tracing (e.g. if crowns are accurately placed on thetooth) and other assessments. Second, the analyzed data, can provide anassessment of the dental image quality, e.g., create signals indicatingthat the dental image is not of high enough quality (e.g., blurry or theteeth structures are overlapping), and that an algorithmic analysis or amanual analysis by a user is not optimal, and can recommend takinganother dental image. Third, the analyzed data can also be employed in aworkflow such as being not visualized but instead (e.g., the area, thetooth number of the detected features, carious lesions on the dentalimage, etc.) can be compared to the diagnosis of the user as it is beinginput into, e.g., practice management software using, e.g., an APIbetween the dental software system and such practice managementsoftware. If, the assessment of the user, e.g., location (tooth numberand position) and/or type of detected feature, is not the same than theanalyzed data, the dental analysis system can send one or morenotifications to the user regarding the event. Furthermore, by mappingthe analyzed data to the associated data of the practice managementsystem, the analyzed data can use time-series analysis and identify theprogress (e.g., the health condition of a patient over period of time).Through such operations, the patient, user of the system, etc. areprovided better information about potential diagnosis and treatmentrecommendations.

In one implementation, the dental analysis system cannot only be usedprospectively but also retrospectively such as by analyzingretrospectively data, e.g., patient records of a dental practice andhospital and matching it with the analyzed diagnoses and treatmentrecommendations of the record, e.g., in the practice management systemor the electronic health record, to estimate the quality of the dentalpractice and analyze if a potential recall of patients is necessary asdental features, e.g., carious lesions or other pathologies, have beenmissed.

The dental analysis system can also provide information such astransactional information to a payor, e.g., the health insurance, whensubmitting a claim. By algorithmically detecting features on the dentalimage and associated dental image information, the system may provide aprobability factor that the diagnosis and recommended treatment of thedentist is accurate and thereby help the payor to detect various typesof events (e.g., potential fraud) and conduct any additional analysis.

Upon one or more features being detected from a representation of theanalyzed data, the detected features can assist in the execution ofseveral functions such as 1) an assistive tool for the user, e.g., thedentist, to support his or her diagnosis and reduce false positive andfalse negative errors, 2) as a second opinion to a patient regardingtheir health conditions and to provide transparency to the diagnosis ofthe user, the dentist, the patient, etc. or 3) as an education tool forcontinuing education of dental professionals, dental students, etc.

The imaging machine, 102, which emits x-ray beams, 104, to an x-raysensor, 106 can be part of an intra-extra oral radiographic imagingmachine (e.g. that produces bitewing and periapical radiographs), anextra-oral radiographic imaging machine (e.g. that produces panoramicx-ray), a dental cone beam computed tomography scan machine for CT-scanscoming from a CT scanner (also called a CBCT-scanner), notradiology-emitting machines such as Positron emission tomography scan(PET-scans) coming from a Positron emission tomography-computedtomography scanner, Magnetic resonance imaging (MRI) scans coming from aMRI scanner, etc.

Referring to FIG. 2 , a computing environment 200 is presented thatincluded a computer system 202, that the user might interact with toview any software output on a display, 204. In an illustrated example,the software user-interface, 206, is presented (e.g. as requested by theuser or automatically presented after a radiographic image is taken). Inthis example, the detected features are displayed using a coloredbounding box 208 that surrounds the detected feature. In one arrangementof this user-interface, the colored box translates to a certainty score,which is decoded in colors, e.g. from green (low confidence) to red(high confidence), that the detected feature is indeed a detectedfeature. In an arrangement, functionalities of this software interface,206, include user selectable icons 210 for executing various functionsuch as deleting and adding detected features. The user can either adddetected features to the radiograph in case the users suggests that thealgorithm missed a detected feature, or he can delete the detectedfeatures of the algorithm, e.g. 208. In one implementation, thecomputing environment 200 is a dental analysis system that a user canprovide feedback about the detected features, for example, by either“agreeing”, “disagreeing”, “clinically validated”, “clinicallyunvalidated”. The input of the user can then be used for additionaltraining data to further improve operations of the machine learningsystem. After carefully reviewing the radiograph using e.g.functionalities such as contrast change, hiding the algorithmicsuggestions and inversion 212, the user can generate a report 214 thatautomatically summaries the algorithmic findings, answers genericquestions to what the detected features mean for the health of thepatient, what treatment recommendations usually are given and gives theuser an way to communicate to the receiver, e.g. patient or other typesof information, recommendations, etc. for his review. The report, 216,can be printed, send via email or transferred by employing one or moreother techniques in any other way to the receiver as provided byselectable icons 218. Furthermore, another selectable button, 220,allows the receiver to easily communicate to the user (e.g., to schedulea follow-up appointment for further treatment or diagnosis, askquestions, etc.). This feature should allow the patient to not miss anyimportant diagnostics or treatment due to a lack for effectivefollow-up.

Referring to FIG. 3 , a computer environment 300 can interact with auser, for example, for viewing detected features (e.g., by interactingwith the user-interface 206, shown in FIG. 2 ). Once a dental image istaken through the sensor 102 (shown in FIG. 1 ), the raw image data getstransferred to a computer system 302 included in the environment 300.From there, either in the raw image data or the post-processed imagedata (e.g., after the sensor's manufacturer's proprietary algorithmshave processed the raw image data), gets exported or otherwisetransferred (e.g. being exported to a memory associated with an ImageRetriever 304 that is executed by the computer system 302. In oneimplementation, the image retriever 304 is a desktop client whichde-identifies the dental image data by deleting or substituting with anon-identifiable replacement all personal health information, e.g. name,date of birth etc. and retains as associated dental image informationonly HIPAA compliant data of the subject (e.g., patient), the image wastaken such as the gender, age, x-ray manufacturer and model of thedental image. Furthermore, the image retriever 304 can check if a dentalimage is a valid image in terms of having a correct image format, is animage which can be processed by the algorithm, and other filtering rulescan apply that the right meta-data etc. contained in the image. Theimage 306 together with its associated dental image information (e.g.age, gender, x-ray modality, sensor, model, other meta-data, etc.), getstransferred over one of more networks (e.g., the internet 308) to afeature detector 310. To provide the functionality of detectingfeatures, the feature detector 310 may use various machine learningtechniques such as deep learning techniques to improve theidentification processes through training the system (e.g., exposemultilayer neural networks to training data, feedback, etc.). Throughsuch machine learning techniques, the feature detector 310 usesartificial intelligence to automatically learn and improve fromexperience without being explicitly programmed. Once trained (e.g., fromx-ray images with and without identified detected features (also calledannotations)), one or more images, representation of images, etc. can beinput into the feature detector 310 to yield an output. The machinelearning may or may not be stored and retrieved at a storage device 316.In this example, access to an identifier 314 is provide through acomputer system 312 (e.g., a server) located at the feature detector310. Further, by returning information about the output (e.g.,feedback), the machine learning technique being used by the identifier314 can use the output as additional training information. Othertraining data can also be provided for further training. By usingincreased amounts of training data (e.g., dental images with and withoutdetected features), feedback data (e.g., data representing userconfirmation, correction or addition of identified detected features),etc., the accuracy of the system can be improved (e.g., to predict imagefeatures). In this illustrated example, the identifier 314 assigns aprobability (e.g. numerical value ranging from 0 to 1, where a largervalue is associated with greater confidence) that a pathology exists toeach pixel in the dental image, which can be post-processed into variousforms (e.g. see FIG. 1 ). An output is provided that represents a set ofconfidence scores for presence of detected image features (e.g., cariouslesions and periapical lucencies), and a conditional probability mapencoding the location of any detected image feature. In one arrangement,an augmented image 318 consisting of the original image, 306, thepixel-wise probability of a detected feature and a graphicalrepresentation, e.g., a bounding box, of the detected feature. Thisaugmented image gets transferred back from the feature detector 310 tothe computer system 302 where the image or portion of the image can beeither displayed in a regular dental image viewer, the user-interface206, other software user-interfaces, etc. The entire dental imagesystem, consisting of the retriever 304 and the feature detector 310,can be either as described above both offline “on premise” on thecomputer system 302 and a connected network, such as the internet 308,or otherwise, completely offline on the computer system 302 or entirelyin the cloud, meaning the internet 308.

Referring to FIG. 4 , one or more techniques may be implemented toidentify detected features in the dental images by executing operationon a computing device (e.g., the computing system 312). For suchtechniques, information may be used from one or more data sources. Forexample, a large data set from many dental practices, hospitals or otherinstitutions who obtain dental images, might be collected in a collectedimage database 404. The identifier 314 is executed by the computersystem 312 (e.g., one or more servers), presents at the feature detector310 (also shown in FIG. 3 ). In this exemplary arrangement, theidentifier 314 includes an image collector 402, which is able to collectimages from the collected image database 404 and the image informationdatabase 406 which has associated dental image information data storedsuch as age, gender, and other image information which may or may not befrequently accessed and used for the identifier 314, regulatory orcomputational reasons both of which are hosted in the storage device316. In this arrangement, such image data may be collected by an imagecollector 402 and stored (e.g., in a collected image database 404) on astorage device 316 for later retrieval. In some arrangements,information associated with images, associated dental image information(e.g., pixel-wise information of the area of the detected features whichwas collected by using the annotator tool, information about thesubject—a patient, the image was taken from, image attributes such asmanufacturers, model, lighting time, etc.) may be provided and stored inan image information database 406. Retrieving the image data (stored indatabase 404) and/or image information (stored in the database 406), amachine learning trainer 408 is provided the data to train a machinelearning inference 412 (Going forward, a “machine learning system” isdefined to consist of both the machine learning trainer 408 and themachine learning inference 412). Various type of data may be used fortraining the system; for example, images (e.g., millions of images) canbe used by the trainer 408. For example, pristine images of dentalimages (e.g., portions of intra-oral bitewing or intra-oral periapicalimages), distorted images of dental images (e.g., synthetically alteredversions), real-world images of dental intraoral cameras (e.g., imagescaptured by individuals in real-world conditions that include one ormore colored pictures of the teeth and gum inside the patient's mouth)may be used to train the machine learning inference 412. For some imagesof dental x-ray images (e.g., images of pristine full mouth series (i.e.a complete set of intraoral X-rays taken of a patients' teeth andadjacent hard tissue (often consisting of four bitewings, eightposterior periapicals, six anterior periapicals), synthetically alteredversions of the same, etc.)), information that identifies each includeddental image feature (e.g., labels) may be provided for training.Alternatively, for some images (e.g., captured under real-worldconditions), identifying information (of included dental image features)may be absent. The trainer 408 can access the image collector data anduse image collector data for training a machine learning model and storeit at the output data base 410. Once trained, the machine learninginference 412 may be provided with input data such as one or more imagesto identify the dental features to detect or if the image quality is toolow is present in the images. For example, after being trained usingpristine, distorted, and real-world images of to be detected imagefeatures, images containing unidentified image features and capturedunder real-world conditions may be input for predicting the contained tobe detected dental features (as illustrated in FIG. 2 ). The identifier314 may output data that represents the predicted dental features or anyother image features (e.g. too low of an image quality or the absence ofsuch dental image features) determined through an analysis of the inputimage. The image information database 406 has corresponding informationfor the images in the collected image database 404 saved. Thisinformation includes, information48271—of the subject (e.g., patient)from whom the x-ray was taken, e.g., age and gender of the individual,the imaging device information, e.g., the type of imaging device (x-ray,CT, etc.), the area/type of image (bitewing or periapical dental image),the hardware model and version and other settings of the imaging devicewhen the image was taken (e.g. all standardized DICOM tags) and theannotations. These annotations may or may not be generated by the datagathering, annotation and educational system as described in FIGS. 9A,9B, and 9C. The images from the collected image database can bepresented to annotators (e.g. dentists, radiologists, other experts ornon-experts) to annotate or mark the region where a feature of interest(e.g. carious lesion which the identifier should be capable ofidentifying) is to be found. The annotator can mark these regions eitherusing a drawing a bounding box close around the feature of interest, bysetting a point into the center of the feature of interest or by drawingan outline around the feature of interest. All these inputs are saved inthe image information database 406 and can serve the trainer as trainingmaterial. In one arrangement, each image does not only get an annotationfrom one individual but several individuals, e.g. three independentannotators, who annotate the same image. All annotations are typicallysaved in the image information database and a software module in theimage collector 402 can automatically combine the multiple annotatorannotations to generate a high-quality annotation. For example, themultiple annotator annotations can be combined in a majority votingsystem (if the two annotators agree on an annotation, the annotationsoverlap with each other for at least 1 pixel or have a certain value of“Intersection over Union”, or a weighted union of all annotation byweighting more to the intersected regions) to define a higher qualityannotation (e.g. 2 of 3 annotators agree on an annotation, it can beconsidered to be very likely a correct annotation, meaning an actualfeature of interest.). This system can be implemented in various wayssuch as having two annotators annotate images and add to data gatheringsystem and then a third annotator serves as a referee and either agreeor disagree with these annotations, and improve the quality of theannotations. By improving the annotations in such a way, the machinelearning trainer can gather a much higher quality of annotations. Forexample, a single value can be output representing existence or absenceof a feature in the entire image. In other arrangements, however, theoutput may be a vector or a matrix, which include a considerable numberof elements (e.g., 1,000,000 elements), one for each pixel, each cariouslesion, etc. A common output matrix can be a heatmap that has the samesize as the input image (i.e., if image is in the size of 1440 by 1920pixel, the matrix will have 1440 rows and 1920 columns) whose elementshave a one-to-one correspondence to the pixels on the input dentalimage. Various types of data may be provided by each element to reflecthow each individual pixel of input image is related to theto-be-detected feature, e.g. carious lesion (a cavity). For example,each element of the matrix may include a floating-point number thatrepresents a level of confidence in detecting the feature, e.g. acarious lesion. In some arrangements, the sum of these element-wisequantities represent a predefined amount (e.g., a value of one) toassist comparing confidence levels and determining which dental imagefeatures, e.g. carious lesions, are closer matches. In this example, theoutput matrix (e.g., with 1440 by 1920 elements) from the machinelearning inference 412 is stored in an output data database 410. Arenderer 414 determines whether a detected image feature (e.g. cariouslesion) are present based on the value of the confidence score and, forany lesion present, generates the coordinates of the lesion boundingbox. The results determined by the renderer 414 (e.g., a list ofpixel-coordinates of the detected feature and its rendered bounding box)can be stored on the storage device 316 (e.g., in an output datadatabase 410) for later retrieval and use. For example, the input images(captured under real-world conditions) and correspondingly identified befurther used to train the machine learning trainer 408 or otherartificial intelligence based systems. The renderer 414 is using thisheatmap and creates an image containing the original radiograph withbounding boxes for any detected feature, the type of detected feature(e.g. carious lesion or periapical radiolucency), and a summarystatement of the number and type of detected features, and a messagestating that the image was analyzed by the software (with link toinstructions/labeling). The renderer 414 can transfer the augmentedimage 318 (or initiate the transfer) either back to the local computerenvironment 302 or visualize the image over an internet-based softwareclient.

Referring to FIG. 5 , a block diagram 500 is presented that provides agraphical representation of the functionality of the machine learninginference 412 and machine learning trainer 408 (shown also in FIG. 4 ).Prior to using the machine learning inference 412 to process an input516 (e.g., a dental image and associated information, e.g., subject age,subject gender, sensor information) to produce an output 520 (e.g., aheat map, which is a matrix of same size as the input image whoseelements represent the level of confidence of the corresponding pixel inthe dental image as the potential detected feature, or a binary maskresulted from thresholding on the aforementioned heatmap, a bounding boxlocalizing the detected feature in the dental image along with a levelof confidence for that bounding box, etc.), the learning system needs tobe trained. The machine learning trainer 408 includes severalcomponents. Various types of training data 502 may be used to preparethe machine learning trainer 408 to identify image features of interestto an end user. For example, dental images in pristine conditions, thecorresponding annotations (bounding box for each feature which should bedetected by the algorithm), and metadata describing some informationabout the image. This data can be in various formats, including but notlimited to image formats, e.g., PNG, TIFF, EXTIFF, databases, and DICOMfile. In some instances, images may be used multiple times for systemtraining to provide the dental image features and associated imageinformation in one or more other forms. From this data, images areextracted by Image Processing Module 504 and resized and cropped to theappropriate shapes. The intensities of the images are normalized tocorrect for acquisition variability of scanners and detectors. Theseimages are then further processed by Image Augmentation 506 to includerandom changes in the images in order to increase variability of theimages, prevent the model to overfit to these images, and finally makethe model more robust. These random changes include randomly flippingthe image upside-down or left-to-right, randomly change the overallbrightness or contrast of the image, randomly crop the image intosmaller size, and randomly make small changes to the annotation. Next,these processed images are fed to a Neural Network Model 508, whichiteratively learns the values for the parameters of the model to predictthe annotation given each image. These values, called weights, and aresaved along with the model architecture at Model Checkpoints 510.Finally, at Model Evaluation 512, the neural network model 508(consisting of architecture and weights) are evaluated on unseen andseparate data from the Training Data 502, and metrics of success isbeing reported. This procedure from Image Augmentation 506 to ModelEvaluation 512 is repeated until the threshold for metrics of successare met. The model that meet these criteria is considered as a TrainedModel 514 and is used in the production and for Machine LearningInference 412. In some arrangements feedback data 522, which can comefrom various sources e.g. detected features from a previous machinelearning model which output has been clinically validated or validatedby another annotator, dental clinician or user, can also be provided tothe machine learning trainer to further improve training. The trainingdata 502 may also include segments of one training image. For example,one image may be segmented into five separate images that focus ondifferent areas of the original image. For prediction operations, aprediction result (e.g., a binary mask, a heatmap or a bounding boxalong with its confidence level) can be attained for each segment and anoverall result determined (e.g., by averaging the individual results) toimprove prediction accuracy. One image may be cropped from the originalimage to focus upon the upper left quadrant of the original image whilethree other segments may be cropped to focus on the upper right, lowerleft, and lower right portions of the original image, respectively. Afifth image segment may be producing by cropping the original image tofocus upon the central portion of the original image. Various sizes andshapes may be used to create these segments; for example, the originalimage may be of a particular size (e.g., 512 by 512 pixels, 1440 by 1920pixels, etc.) while the segments are of lesser size (e.g., 256 by 256pixels). In one arrangement, which is called active learning, afterinitial training with the first set of annotated dental images (e.g.,5,000 dental images), for each new dental image, which has not beenannotated (each remaining of the 10,000 dental images for instance),operations are executed (by the identifier 314) to determine the mostvaluable images for further annotation and training.

In production phase, the Input 516 is typically an image (or a set ofimages) without any annotation. This image is usually processed with thesame Image Preprocessing Module 504 that is used in Machine LearningTrainer 408. Then, without any further processing, the image is fed tothe Trained Model 514 and the model predict the target output (e.g., abounding box, a heatmap, or a binary mask) for any present detectedfeature. These intermediate outputs are put together and superimposed onthe original input image in Postprocessing 518 and results in the Output520 that can be rendered on the users' workstation.

To train the machine learning trainer 408 and implement algorithms intothe machine learning inference 412, one or more machine learningtechniques may be employed. For example, supervised learning techniquesmay be implemented in which training is based on a desired output thatis known for an input. Supervised learning can be considered an attemptto learn a nonlinear function that maps inputs to outputs and thenestimate outputs for previously unseen inputs (a newly introducedinput). Depending on the desired output, these supervised learningmethods learn different nonlinear functions and perform different tasks.The output can be just a text or alarm that signal the presence orabsence of a lesion or any other feature of interest like number ofteeth. This task is being done by classification methods, but if theoutput is a continuous value like the size of a cavity, regressionmethods are being used. On the other hand, the output can be a visualfeature, like the delineation of a tooth or a lesion or just a box thatincludes that tooth or lesion. Using exact delineation of a feature ofinterest as the output, we can employ segmentation algorithms to performthe supervised learning task. When boxes that are superimposed on theinput images, called bounding boxes, are used as the desired output, theobject detection algorithms are employed. Unsupervised learningtechniques may also be employed in which training is provided from knowninputs but unknown outputs. Dimensionality reduction methods are examplesuch techniques that tries to find patterns in the data and can create amore compact representation of the image. This compact representationthen can be correlated to certain features of interest. Reinforcementlearning techniques may also be used in which the system can beconsidered as learning from consequences of actions taken (e.g., inputsvalues are known). This can be mainly used for dental treatmentplanning, like orthodontics treatment, to learning the optimal treatmentstrategy. In some arrangements, the implemented technique may employ twoor more of these methodologies. In some arrangements, neural networktechniques may be implemented using the data representing the images(e.g., a matrix of numerical values that represent visual elements suchas pixels of an image, etc.) to invoke training algorithms forautomatically learning the images and related information. Such neuralnetworks typically employ a number of layers. Once the layers and numberof units for each layer is defined, weights and thresholds of the neuralnetwork are typically set to minimize the prediction error throughtraining of the network. Such techniques for minimizing error can beconsidered as fitting a model (represented by the network) to trainingdata. By using the image data (e.g., attribute vectors), a function maybe defined that quantifies error (e.g., a squared error function used inregression techniques). By minimizing error, a neural network may bedeveloped that is capable of determining attributes for an input image.One or more techniques may be employed by the machine learning system(the machine learning trainer 408 and machine learning system 412), forexample, backpropagation techniques can be used to calculate the errorcontribution of each neuron after a batch of images is processed.Stochastic gradient descent, also known as incremental gradient descent,can be used by the machine learning system as a stochastic approximationof the gradient descent optimization and iterative method to minimize aloss function. Other factors may also be accounted for during neutralnetwork development. For example, a model may too closely attempt to fitdata (e.g., fitting a curve to the extent that the modeling of anoverall function is degraded). Such overfitting of a neural network mayoccur during the model training and one or more techniques may beimplements to reduce its effects. Other types of artificial intelligencetechniques may be employed about the identifier 314 (shown in FIG. 3 andFIG. 4 ). For example, the machine learning inference 412 and machinelearning trainer 408 can use neural networks such as a generativeadversarial networks (GANs) in its machine learning architecture (e.g.,an unsupervised machine learning architecture). In general, a GANincludes a generator neural network, a different specific ximplementation kind of the Image Augmentation 506, that generates data(e.g., different versions of the same image by flips, inversions,mirroring etc.) that is evaluated by a discriminator neural network, aspecific type of the Neural Network Model 508, for authenticity (e.g.,to identify the dental images). In other words, the discriminator neuralnetwork, a specific type of the Neural Network Model 508, attempts toidentify the detected feature included in the augmented image (e.g., adistorted version of a dental image) provided by the generator, adifferent specific implementation of the Image Augmentation 506. Variousimplementations for GAN generators and discriminators may be used; forexample, the discriminator neural network, a specific type of the NeuralNetwork Model 508, can use a convolutional neural network thatcategorizes input images with a binomial classifier that labels theimages as genuine or not. The generator neural network, a differentspecific implementation of the Image Augmentation 506, can use aninverse convolutional (or deconvolutional) neural network that takes avector of random noise and upsamples the vector data to an image toaugment the image.

Other forms of artificial intelligence techniques may be used by themachine learning trainer 408 and machine learning inference 412. Forexample, to process information (e.g., images, image representations,etc.) to identify detected features of the x-ray image, such aspotential cavities and periapical radiolucencies, the architecture mayemploy decision tree learning that uses one or more decision trees (as apredictive model) to progress from observations about an item(represented in the branches) to conclusions about the item's target(represented in the leaves). In some arrangements, random forests orrandom decision forests are used and can be considered as an ensemblelearning method for classification, regression and other tasks. Suchtechniques generally operate by constructing a multitude of decisiontrees at training time and outputting the class that is the mode of theclasses (classification) or mean prediction (regression) of theindividual trees. Support vector machines (SVMs) can be used that aresupervised learning models with associated learning algorithms thatanalyze data used for classification and regression analysis. Ensemblelearning systems may also be used for detecting features in dentalimages in which multiple system members independently arrive at aresult. The ensemble typically comprises not only algorithms withdiverse architectures, but also algorithms trained on multipleindependent data sets. In one arrangement, a convolutional neuralnetwork architecture can be used that is based on U-Net to perform imagesegmentation to identify detected features, e.g. radiolucent lesions andcarious lesions on the dental x-ray images. This implementation of thenetwork uses batch-normalization after each convolutional layer has atunable depth. The network parameters (weights) are trained using theJaccard Index metric as a loss function, where true positive, falsepositive and false negative counts are measured across all images in abatch/mini-batch. The algorithm assigns a probability (e.g. numberranging from 0 to 1, where a larger value is associated with greaterconfidence) that a pathology exists to each pixel in the x-ray image,which can be post-processed into various non-graphical or graphicalforms (e.g. see 208). The algorithm is trained using data augmentationof the images and ground truth regions, for example one or more ofrotations, scaling, random crops, translations, image flips, and elastictransformations; the amount of augmentation for each transformation istuned to optimize performance of the algorithm on the available data.System members can be of the same type (e.g., each is a decision treelearning machine, etc.) or members can be of different types (e.g., oneDeep CNN system, one SVM system, one decision tree system, etc.). Uponeach system member determining a result, a majority vote among thesystem members is used (or other type of voting technique) to determinean overall prediction result. In some arrangements, one or moreknowledge-based systems such as an expert system may be employed. Ingeneral, such expert systems are designed by solving relatively complexproblems by using reasoning techniques that may employ conditionalstatements (e.g., if-then rules). In some arrangements such expertsystems may use multiple systems such as a two sub-system design, inwhich one system component stores structured and/or unstructuredinformation (e.g., a knowledge base) and a second system componentapplies rules, etc. to the stored information (e.g., an inferenceengine) to determine results of interest (e.g., select images likely tobe presented).

System variations may also include different hardware implementationsand the different uses of the system hardware. For example, multipleinstances of the machine learning system identifier 314 may be executedthrough the use of a single graphical processing unit (GPU). In such animplementation, multiple system clients (each operating with one machinelearning system) may be served by a single GPU. In other arrangements,multiple GPU's may be used. Similarly, under some conditions, a singleinstance of the machine learning system may be capable of servingmultiple clients. Based upon changing conditions, multiple instances ofa machine learning system may be employed to handle an increasedworkload from multiple clients. For example, environmental conditions(e.g., system throughput), client-based conditions (e.g., number ofrequests received per client), hardware conditions (e.g., GPU usage,memory use, etc.) can trigger multiple instances of the system to beemployed, increase the number of GPU's being used, etc. Similar totaking steps to react to an increase in processing capability,adjustments can be made when less processing is needed. For example, thenumber of instances of a machine learning system being used may bedecreased along with the number of GPU's needed to service the clients.Other types of processors may be used in place of the GPU's or inconcert with them (e.g., combinations of different types of processors).For example, central processing units (CPU's), processors developed formachine learning use (e.g., an application-specific integrated circuit(ASIC) developed for machine learning and known as a tensor processingunit (TPU)), etc. may be employed. Similar to GPU's one or more modelsmay be provided by these other types of processors, either independentlyor in concert with other processors.

FIG. 6 illustrates an exemplary meta-architecture for the Neural NetworkModel 508, according to various arrangements. This meta-architecture isfor object detection with region proposal network. The Training Data 502consists of Image 602 and Annotations 604 (meaning the pixel coordinateinformation of the to be detected features and, in an arrangement,additional dental image associated information such as image's subjectinformation such as age, gender, etc. and other parameters of the imagesuch as contrast, width, pixel-depth, etc.). Each of these componentsare separately processed and prepared by the Image Processing Module 504to obtain Processed Image 606 and Processed Annotations 608, which areready to be fed to the model. However, usually we further process andaugment them (e.g., only during training) to obtain Augmented Image 610and Augmented Annotations 612. These are usually the two inputs to the(deep) neural network model 508. Augmented Image 610 go throughConvolutional Neural Network 614, which consists of many convolutionallayers (e.g. 101 layers). The output of this network is bifurcated to beused in two parallel tasks. One is extracting the learned features andcreate Feature Maps 616 and the other is used by a Region ProposalNetwork 618 to propose bounding boxes with different shape and sizeassociated to each target class. Then these bounding boxes are mergedwith the feature maps through region of interest (ROI) Pooling 620, andcreate region of interests (ROI's), which are potential candidates fordetection. These ROI's go through Fully Connected Layers 622, which aresocial types of neural network layers where components are denselyconnected to the components of previous layer. Next, this output isbifurcated with independent Fully Connected Layers 622 forclassification of type of the detection, using a Softmax Classifier 624,and tightening the bounding boxes using a Bounding-box Regressor 626.The model is trained by combining the output of the Softmax Classifier624 and the Bounding Box Regressor 626 to build a Loss Function 628 andminimize the value of this Loss Function 628 via one or moreoptimization methods, such as stochastic gradient decent. In otherinstances, other meta-architectures can be employed that may or may notrely on region proposal. These meta-architectures include but notlimited to Single Shot Multi Box Detector, YOLO9000, YOLOv2, YOLOv3,Feature Pyramid Networks, RetinaNet.

FIG. 7 illustrates an exemplary network architecture for theConvolutional Neural Network 614, according to various arrangements. Inthis example, two special neural network's blocks are employed: anIdentity Block 702 and a Cony Block 704. These blocks are used in asequence to build a deep neural network architecture, referred to asResNet Architecture 706. There are various ways to build a deep neuralnetwork architecture, either putting together these building blocks tocreate a customized network or use pre-existing architectures likeAlexNet, VGG, DenseNet, InceptionV3, Xception, MobileNet, NASNet, etc.The number of parameters in these architectures can vary from fewhundreds to hundreds of millions of parameters.

FIG. 8 shows the instructions to build the convolutional neural networkarchitecture. Section 802 shows the detailed instruction of skipconnection building blocks (Identity Block 702 and Cony Block 704) wherethe input bifurcate goes through a sequence of convolutional and batchnormalization layer with ReLU activations on one branch and just oneconvolution and batch normalization (Cony Block 704) or no operation(Identity Block 702) in other branch. These branches are merged at theend of the block by adding them together. Section 804 includesinstructions for ResNet Architecture 706, which uses the aforementionedblocks along with convolutional, batch-normalization, max pooling, andaverage pooling later to build a convolutional neural network model.

Referring to FIGS. 9A, 9B, and 9C, a flowchart of screenshots presents adata gathering, annotation and educational system. The flowchartrepresents the gathering of various types of data from the user of thesystem, the subject of the system (e.g. dental images) to collecttraining data for the operations executed at the Feature Detector 310,locally at the computer system 302 or in the internet, etc. as describedwith respect to FIG. 1-7 , to train dental professionals (e.g. students,dentists, etc.) and to conduct research studies. In an administrativeinterface 902, an administrator (e.g. professor, manager etc.) canselect images, types of detected features he would like theannotator/user to annotate/mark, questions he might want to ask andother characteristics of the tasks (e.g. the type of imaging). He canthen send out the task to the annotator/user, (e.g. dentists, student,any other human being or computer), who can see an overview of hisperformance on an annotator/user-dashboard 904. In this dashboard theannotator/user can access “Annotations tasks”, “Training/learningtasks”, and “quiz tasks”. Once selected e.g. the annotation task, theuser is transferred to the annotator tool 906. This module has x-rayviewer capabilities (e.g., inversion, change of contrast, etc.) and canbe used to diagnose the dental image and input an “annotation”. Forexample, the user can provide a e.g. bounding box, or an outline aroundthe image feature that is detected by the identifier 314 (shown in FIG.3 ). In an arrangement of the system, after annotating the image, thenext module can be the comparison interface 908. This interface 908compares the annotation of the user, with either the algorithmicanalysis of such image or any other annotation of another user or agroup of other users. The interface 908 automatically detects if thereis an agreement between the user and this algorithmic/other userannotation (e.g. by an overlap of minimum one pixel). If there is nooverlap, the user can decide if he agrees with the algorithmic/otheruser annotation or not. All this input data (e.g. the pixel-wiselocation of the annotation of the user, the location of thealgorithmic/other user annotation, the agreement or disagreement betweenthe two, the use of contrast, inversion, time spend on the dental imageetc.) gets saves in the backend. One or more comparison techniquesautomatically calculates performance metrices for the user to have anexpert. It computes the false positive, true positive and true negativevalue of the user's annotations based on some “ground-truth” standardwhich has been previously defined. This ground-truth standard can eitherbe based on an expert (e.g. professor) who annotated the imagespreviously, based on clinical studies which assessed the patients of thedental imaging, medical records, a combination of many peopleannotations or any other mean. Comparing the user's annotations againstthis ground-truth, the other user's annotations and the algorithmicannotation (output), allows the system to compute a variety ofperformance metrices such as how “accurate” in terms of specificity andsensitivity the user is compared to other users, other experts orcompared to the algorithm. This data 912 can be output and used by thedental software system in a variety of ways. In an arrangement, not allimages that are annotated need to have a pre-defined ground-truth.Furthermore, by accumulating several user's annotation for the sameimage, these annotations can be groups using specific clusteringalgorithms, calculating agreement rates and providing us with bettertraining data for the identifier 314. At the same time, it allowsmedical professionals (e.g. dental students, dentists, radiologists) toclose the often-missing feedback loop in medicine. Often you do not knowif an assessment was actually correct or not as you do not see thepatient again or the diagnosis/treatment does not allow to observe thecounter-factual. This system is an interactive training system to betterlearn to diagnose dental and other types of medical imaging and at thesame time allows to collect valuable data to train artificialintelligence software.

Referring to FIG. 10 , a flowchart 1000 represents operations of theidentifier 314 (shown in FIG. 3 ). Operations of the identifier 314 aretypically executed by a single computing device (e.g., the computersystem 312); however, operations of the identifier may be executed bymultiple computing devices. Along with being executed at a single site,execution of operations may be distributed among two or more locations.

Operations of the identifier include receiving 1002 data representingone or more images of dental information associated with a patient. Forexample, one or multiple radiographic images may be received thatcontain dental information about a patient or multiple patients (e.g.,jaw and teeth images). Operations also include adjusting 1004 the datarepresenting the one or more images of dental information into apredefined format. For example, raw imagery may be processed to beingrepresented in a DICOM format or other time of image format. Adjustingthe data includes adjusting one or more visual parameters associatedwith the one or more images of dental information. For example, imagery,information associated with the images, etc. may be filtered orprocessed in other manners. Operations also include using 1006 a machinelearning system to determine a confidence score for one or more portionsof the one or more images of dental information. For example, aconfidence score (e.g., having a numerical value from 0 to 1) can beassigned to each pixel associated with a dental image that reflects thepresence of a feature (e.g., e.g., carious lesions and periapicallucencies). Operations also include producing 1008 a representation ofthe determined confidence scores to identify one or more detectedfeatures present in the one or more images of dental information. Forexample, graphical representation (e.g., colored bounding boxes) may bepresented on a graphical interface to represent the certainty score andalert the viewer to the detected features.

FIG. 11 shows an example of example computer device 1100 and examplemobile computer device 1150, which can be used to implement thetechniques described herein. For example, a portion or all of theoperations of the identifier 314 (shown in FIG. 3 ) may be executed bythe computer device 1100 and/or the mobile computer device 1150.Computing device 1100 is intended to represent various forms of digitalcomputers, including, e.g., laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. Computing device 1150 is intended to representvarious forms of mobile devices, including, e.g., personal digitalassistants, tablet computing devices, cellular telephones, smartphones,and other similar computing devices. The components shown here, theirconnections and relationships, and their functions, are meant to beexamples only, and are not meant to limit implementations of thetechniques described and/or claimed in this document.

Computing device 1100 includes processor 1102, memory 1104, storagedevice 1106, high-speed interface 1108 connecting to memory 1104 andhigh-speed expansion ports 1110, and low speed interface 1112 connectingto low speed bus 1114 and storage device 1106. Each of components 1102,1104, 1106, 1108, 1110, and 1112, are interconnected using variousbusses, and can be mounted on a common motherboard or in other mannersas appropriate. Processor 1102 can process instructions for executionwithin computing device 1100, including instructions stored in memory1104 or on storage device 1106 to display graphical data for a GUI on anexternal input/output device, including, e.g., display 1116 coupled tohigh speed interface 1108. In other implementations, multiple processorsand/or multiple busses can be used, as appropriate, along with multiplememories and types of memory. Also, multiple computing devices 1100 canbe connected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

Memory 1104 stores data within computing device 1100. In oneimplementation, memory 1104 is a volatile memory unit or units. Inanother implementation, memory 1104 is a non-volatile memory unit orunits. Memory 1104 also can be another form of computer-readable medium(e.g., a magnetic or optical disk. Memory 1104 may be non-transitory.)

Storage device 1106 is capable of providing mass storage for computingdevice 1100. In one implementation, storage device 1106 can be orcontain a computer-readable medium (e.g., a floppy disk device, a harddisk device, an optical disk device, or a tape device, a flash memory orother similar solid state memory device, or an array of devices, such asdevices in a storage area network or other configurations.) A computerprogram product can be tangibly embodied in a data carrier. The computerprogram product also can contain instructions that, when executed,perform one or more methods (e.g., those described above.) The datacarrier is a computer- or machine-readable medium, (e.g., memory 1104,storage device 1106, memory on processor 1102, and the like.)

High-speed controller 1108 manages bandwidth-intensive operations forcomputing device 1100, while low speed controller 1112 manages lowerbandwidth-intensive operations. Such allocation of functions is anexample only. In one implementation, high-speed controller 1708 iscoupled to memory 1104, display 1116 (e.g., through a graphics processoror accelerator), and to high-speed expansion ports 1110, which canaccept various expansion cards (not shown). In the implementation,low-speed controller 1112 is coupled to storage device 1106 andlow-speed expansion port 1114. The low-speed expansion port, which caninclude various communication ports (e.g., USB, Bluetooth®, Ethernet,wireless Ethernet), can be coupled to one or more input/output devices,(e.g., a keyboard, a pointing device, a scanner, or a networking deviceincluding a switch or router, e.g., through a network adapter.)

Computing device 1100 can be implemented in a number of different forms,as shown in the figure. For example, it can be implemented as standardserver 1120, or multiple times in a group of such servers. It also canbe implemented as part of rack server system 1124. In addition or as analternative, it can be implemented in a personal computer (e.g., laptopcomputer 1122.) In some examples, components from computing device 1100can be combined with other components in a mobile device (not shown),e.g., device 1150. Each of such devices can contain one or more ofcomputing device 1100, 1150, and an entire system can be made up ofmultiple computing devices 1100, 1150 communicating with each other.

Computing device 1150 includes processor 1152, memory 1164, aninput/output device (e.g., display 1154, communication interface 1166,and transceiver 1168) among other components. Device 1150 also can beprovided with a storage device, (e.g., a microdrive or other device) toprovide additional storage. Each of components 1150, 1152, 1164, 1154,1166, and 1168, are interconnected using various buses, and several ofthe components can be mounted on a common motherboard or in othermanners as appropriate.

Processor 1152 can execute instructions within computing device 1150,including instructions stored in memory 1164. The processor can beimplemented as a chipset of chips that include separate and multipleanalog and digital processors. The processor can provide, for example,for coordination of the other components of device 1150, e.g., controlof user interfaces, applications run by device 1150, and wirelesscommunication by device 1150.

Processor 1152 can communicate with a user through control interface1158 and display interface 1156 coupled to display 1154. Display 1154can be, for example, a TFT LCD (Thin-Film-Transistor Liquid CrystalDisplay) or an OLED (Organic Light Emitting Diode) display, or otherappropriate display technology. Display interface 1156 can compriseappropriate circuitry for driving display 1154 to present graphical andother data to a user. Control interface 1158 can receive commands from auser and convert them for submission to processor 1152. In addition,external interface 1162 can communicate with processor 1142, so as toenable near area communication of device 1150 with other devices.External interface 1162 can provide, for example, for wiredcommunication in some implementations, or for wireless communication inother implementations, and multiple interfaces also can be used.

Memory 1164 stores data within computing device 1150. Memory 1164 can beimplemented as one or more of a computer-readable medium or media, avolatile memory unit or units, or a non-volatile memory unit or units.Expansion memory 1174 also can be provided and connected to device 1150through expansion interface 1172, which can include, for example, a SIMM(Single In Line Memory Module) card interface. Such expansion memory1174 can provide extra storage space for device 1150, or also can storeapplications or other data for device 1150. Specifically, expansionmemory 1174 can include instructions to carry out or supplement theprocesses described above, and can include secure data also. Thus, forexample, expansion memory 1174 can be provided as a security module fordevice 1150, and can be programmed with instructions that permit secureuse of device 1150. In addition, secure applications can be providedthrough the SIMM cards, along with additional data, (e.g., placingidentifying data on the SIMM card in a non-hackable manner.)

The memory can include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in a data carrier. The computer program productcontains instructions that, when executed, perform one or more methods,e.g., those described above. The data carrier is a computer- ormachine-readable medium (e.g., memory 1164, expansion memory 1174,and/or memory on processor 1152), which can be received, for example,over transceiver 1168 or external interface 1162.

Device 1150 can communicate wirelessly through communication interface1166, which can include digital signal processing circuitry wherenecessary. Communication interface 1166 can provide for communicationsunder various modes or protocols (e.g., GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.)Such communication can occur, for example, through radio-frequencytransceiver 1168. In addition, short-range communication can occur,e.g., using a Bluetooth®, WiFi, or other such transceiver (not shown).In addition, GPS (Global Positioning System) receiver module 1170 canprovide additional navigation- and location-related wireless data todevice 1150, which can be used as appropriate by applications running ondevice 1150. Sensors and modules such as cameras, microphones,compasses, accelerators (for orientation sensing), etc. may be includedin the device.

Device 1150 also can communicate audibly using audio codec 1160, whichcan receive spoken data from a user and convert it to usable digitaldata. Audio codec 1160 can likewise generate audible sound for a user,(e.g., through a speaker in a handset of device 1150.) Such sound caninclude sound from voice telephone calls, can include recorded sound(e.g., voice messages, music files, and the like) and also can includesound generated by applications operating on device 1150.

Computing device 1150 can be implemented in a number of different forms,as shown in the figure. For example, it can be implemented as cellulartelephone 1180. It also can be implemented as part of smartphone 1182,personal digital assistant, or other similar mobile device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor. Theprogrammable processor can be special or general purpose, coupled toreceive data and instructions from, and to transmit data andinstructions to, a storage system, at least one input device, and atleast one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms machine-readable medium andcomputer-readable medium refer to a computer program product, apparatusand/or device (e.g., magnetic discs, optical disks, memory, ProgrammableLogic Devices (PLDs)) used to provide machine instructions and/or datato a programmable processor, including a machine-readable medium thatreceives machine instructions.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a device fordisplaying data to the user (e.g., a CRT (cathode ray tube) or LCD(liquid crystal display) monitor), and a keyboard and a pointing device(e.g., a mouse or a trackball) by which the user can provide input tothe computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be a form of sensory feedback (e.g., visual feedback, auditoryfeedback, or tactile feedback); and input from the user can be receivedin a form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a backend component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a frontend component (e.g., a client computerhaving a user interface or a Web browser through which a user caninteract with an implementation of the systems and techniques describedhere), or a combination of such back end, middleware, or frontendcomponents. The components of the system can be interconnected by a formor medium of digital data communication (e.g., a communication network).Examples of communication networks include a local area network (LAN), awide area network (WAN), and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

In some implementations, the engines described herein can be separated,combined or incorporated into a single or combined engine. The enginesdepicted in the figures are not intended to limit the systems describedhere to the software architectures shown in the figures.

A number of embodiments have been described. Nevertheless, it will beunderstood that various modifications can be made without departing fromthe spirit and scope of the processes and techniques described herein.In addition, the logic flows depicted in the figures do not require theparticular order shown, or sequential order, to achieve desirableresults. In addition, other steps can be provided, or steps can beeliminated, from the described flows, and other components can be addedto, or removed from, the described systems. Accordingly, otherembodiments are within the scope of the following claims.

What is claimed is:
 1. A computing device implemented method comprising:receiving data representing one or more images of dental informationassociated with a patient; adjusting the data representing the one ormore images of dental information into a predefined format, whereinadjusting the data includes adjusting one or more visual parametersassociated with the one or more images of dental information; using amachine learning system to determine a confidence score for one or moreportions of the one or more images of dental information; and producinga representation of the determined confidence scores to identify one ormore detected features present in the one or more images of dentalinformation.
 2. The computing device implemented method of claim 1,further comprising: transferring data representative of the one or moreimages of dental information associated with the patient to one or morenetworked computing devices for statistical analysis.
 3. The computingdevice implemented method of claim 1, wherein the machine learningsystem employs a convolution neural network.
 4. The computing deviceimplemented method of claim 1, wherein the machine learning is trainedwith dental imagery and associated annotations.
 5. The computing deviceimplemented method of claim 1, wherein one or more annotations areproduced for each of the images of dental information.
 6. The computingdevice implemented method of claim 1, wherein the one or more detectedfeatures include a radiolucent lesion or an opaque lesion.
 7. Thecomputing device implemented method of claim 1, wherein the producedrepresentation includes a graphical representation that is presentableon a user interface of the computing device.
 8. The computing deviceimplemented method of claim 1, wherein the produced representation isused for a diagnosis and treatment plan.
 9. The computing deviceimplemented method of claim 8, wherein an alert or recommendation isproduced by using the produced representation for the diagnosis andtreatment plan.
 10. A system comprising: a computing device comprising:a memory configured to store instructions; and a processor to executethe instructions to perform operations comprising: receiving datarepresenting one or more images of dental information associated with apatient; adjusting the data representing the one or more images ofdental information into a predefined format, wherein adjusting the dataincludes adjusting one or more visual parameters associated with the oneor more images of dental information; using a machine learning system todetermine a confidence score for one or more portions of the one or moreimages of dental information; and producing a representation of thedetermined confidence scores to identify one or more detected featurespresent in the one or more images of dental information.
 11. The systemof claim 10, further comprising: transferring data representative of theone or more images of dental information associated with the patient toone or more networked computing devices for statistical analysis. 12.The system of claim 10, wherein the machine learning system employs aconvolution neural network.
 13. The system of claim 10, wherein themachine learning is trained with dental imagery and associatedannotations.
 14. The system of claim 10, wherein one or more annotationsare produced for each of the images of dental information.
 15. Thesystem of claim 10, wherein the one or more detected features include aradiolucent lesion or an opaque lesion.
 16. The system of claim 10,wherein the produced representation includes a graphical representationthat is presentable on a user interface of the computing device.
 17. Thesystem of claim 10, wherein the produced representation is used for adiagnosis and treatment plan.
 18. The system of claim 17, wherein analert or recommendation is produced by using the produced representationfor the diagnosis and treatment plan.
 19. One or more computer readablemedia storing instructions that are executable by a processing device,and upon such execution cause the processing device to performoperations comprising: receiving data representing one or more images ofdental information associated with a patient; adjusting the datarepresenting the one or more images of dental information into apredefined format, wherein adjusting the data includes adjusting one ormore visual parameters associated with the one or more images of dentalinformation; using a machine learning system to determine a confidencescore for one or more portions of the one or more images of dentalinformation; and producing a representation of the determined confidencescores to identify one or more detected features present in the one ormore images of dental information.
 20. The computer readable media ofclaim 19, further comprising: transferring data representative of theone or more images of dental information associated with the patient toone or more networked computing devices for statistical analysis. 21.The computer readable media of claim 19, wherein the machine learningsystem employs a convolution neural network.
 22. The computer readablemedia of claim 19, wherein the machine learning is trained with dentalimagery and associated annotations.
 23. The computer readable media ofclaim 19, wherein one or more annotations are produced for each of theimages of dental information.
 24. The computer readable media of claim19, wherein the one or more detected features include a radiolucentlesion or an opaque lesion.
 25. The computer readable media of claim 19,wherein the produced representation includes a graphical representationthat is presentable on a user interface of the computing device.
 26. Thecomputer readable media of claim 19, wherein the produced representationis used for a diagnosis and treatment plan.
 27. The computer readablemedia of claim 27, wherein an alert or recommendation is produced byusing the produced representation for the diagnosis and treatment plan.